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\0 ■ Abstract 

o ; 

5 ■ We consider a communication system in which the outputs of a Markov source are 

encoded and decoded in real-time by a finite memory receiver, and the distortion 
measure does not tolerate delays. The objective is to choose designs, i.e. real-time 
^ ■ encoding, decoding and memory update strategies that minimize a total expected 

distortion measure. This is a dynamic team problem with non-classical information 
structure [Zj. We use the structural results of Q to develop a sequential decom- 
position for the finite and infinite horizon problems. Thus, we obtain a systematic 
methodology for the determination of jointly optimal encoding decoding and mem- 
c/3 ' ory update strategies for real-time point-to-point communication systems. 

Keywords: Real-time communication, finite-delay communication, zero-delay communi- 
cation, joint source-channel coding, Markov decision theory 
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1 Introduction 



> 

O ' Real-time communication problems arise in controlled decentralized systems where in- 

formation must be exchanged between various nodes of the system and decisions based 
V/-^ ! on the communicated information must be made in real-time. Such systems include 

O I QoS (delay) requirements and distributed routing in wired, wireless and sensor networks, 

^ ■ traffic fiow control in transportation networks, resource allocation and consensus in par- 

tially synchronous systems and decentralized resource allocation problems in economic 
. systems. 

', We consider point-to-point real-time communication system as shown in Figure HJ 

_c3 I which is the simplest system of this class. A better understanding of this case is needed 

before generalizing to multi-terminal systems. In the system under consideration, the 
outputs of a Markov source are encoded in real-time into a sequence of random variables. 
This sequence is transmitted through a discrete memoryless channel (DMC) to a receiver 
with finite memory. At each time instant t, using the current channel output and its 
current memory content, the receiver updates its memory and estimates the source output 
at t. The system designer has to choose real-time encoding, decoding and memory update 
rules that minimize an expected total distortion. 

Real-time or finite-delay communication problems have been considered in the past. 
For an extensive literature survey we refer the reader to Here we will only refer 

to the papers most relevant to our philosophy and approach. Real-time encoding and 
decoding with limited memory make the standard Information theoretic techniques in- 
appropriate for this problem. Most of the results of Information theory are based on 
some form of the law of large numbers, which becomes applicable only when we consider 
sufficiently long sequences. This problem does not have enough structure to use encoding 
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Figure 1: Real-Time Communication System 



and decoding of typical sequences. Hence, we consider a decision theoretic approach to 
the problem. 

Decision theoretic approaches to real-time communication similar in spirit to ours have 
appeared in [IH [H Ej |4] . Real-time communication problems for noiseless channel were 
studied in [Hill]. Real-time encoding decoding problems for a noisy channel and noise- 
less feedback were studied in [SI El- These problems share a common feature that at 
every stage the encoder has perfect knowledge of the information available to the de- 
coder/receiver. The case of a noisy channel and no feedback does not share this feature. 
Real-time communication through noisy channels and no feedback was investigated in jlj 
and structural results for optimal real-time encoding and decoding strategies were ob- 
tained. However, to the best of our knowledge, the problem of obtaining jointly optimal 
real-time encoders, decoders and memory update rules has not been considered by anyone 
so far. In this paper we present a methodology for this joint optimization. We present 
the key ideas and fundamental results here, and refer the reader to [3j for details and 
extensions. 

The remainder of the paper is organized as follows. In Section El we formally define 
the problem, in Section |2l we restate the structural results of |3], in Section |31 we present 
the joint optimization of encoder, decoder and memory update. In Sections El and IHl we 
consider the finite and infinite horizon time homogeneous cases. We discuss some salient 
points in Section O and conclude in Section (HI 

Notation: When using English letters to represent a variable, we use the standard 
notation of using uppercase letters (X, Y, Z) for denoting random variables and lowercase 
letters for denoting their realization {x,y,z). While representing a function of random 
variables as a random variable {Pm, Py,m), we use a tilde above the variable to denote its 
realization {Pm, Py,m)- When using Greek letters to represent a random variable (tt, ip), 
we use a tilde above the variable to denote its representation (tt,*^). We also use the 
standard short-hand notation of x* to represent the sequence Xg, ■ ■ ■ ,Xt ^ x\\s abbreviated 
to X* and similar notation for random variables and functions. 

2 Problem Formulation 

We now give a formal description of the problem under consideration. Consider a discrete 
time communication system shown in Figure [H A first order Markov source produces a 
random sequence Xi, . . . , Xt. For simplicity of exposition we assume that Xt belongs to 
a finite alphabet A" = { 1, . . . , \X\ }. 

At each stage t, the encoder can transmit a symbol Zt taking values in a finite alphabet 
Z = {1, . . . ,\Z\} . This encoded symbol must be generated in real-time, i.e., 

Zt = Ct{X,,...,Xt), t = l,...,T, (1) 
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and transmitted through a 1 2^ | -input 1 3^ | -output discrete memoryless channel producing 
the sequence { Fi, . . . , 1^ }, with each Yt belonging to an alphabet y = {1,...,|3^|}. 
The transition probabilities of the channel is given by 

Pr {yt I x\ z\ y'-^) = Pr {yt \ Zt) = Pt{yt, Zt). (2) 

At the receiver, the most that could be accessible at stage t is the subsequence Fi, . . . , 1^. 
However, we assume that the receiver has a memory of log2 \M.\ bits. So, after some 
time, all the past observations can not be stored and the receiver must selectively shed 
information. We model this by assuming that the contents of the memory belong to a 
finite alphabet M = {I, . . . ,\M\}. The memory is arbitrarily initialized with Mq = 1 
and then updated at each stage according to the rule 

Mt = lt{Yt,Mt.{), t = l,...,T-l. (3) 

The objective of the decoder is to generate an estimate of the source output in real- 
time. This estimate Xt has to be generated from the present channel output Yt and the 
memory contents M(_i, by some decoding rule, i.e., 

Xt = gt{Yt,Mt.^), t=l,...,T. (4) 

The performance of the system is defined by way of a sequence of distortion functions. 
For each t, pt : X x X ^ [0,oo). is given. Then, pt{Xt,Xt) measures the distortion at 
stage t. 

A choice (c, g, I) of decision rules for all stages is called a design, where c = (ci, . . . , ct), 
g = (gi, . . . , gx) and I = (li, . . . , It-i)- The performance of a design is quantified by the 
expected distortion under that design, which is given by 



J(c,<7,0 = E<^ Y^PtiXt.Xt 



t=i 



c,g,l > . (5) 



The optimization problem that we consider is as follows: 

Problem 1. Assume that the encoder and the receiver know the statistics of the source 
(i.e. PMF of Xi and the transition probabilities Pxt+i\Xt); the channel transition matrix 
Ft, the distortion function Pt{-,-) o-i^d time horizon T. Choose a design {c*,g*,l*) that 
is optimal with respect to the performance criterion of (jSJ, i.e., 

J{c\g\r)=T^ mill J{c,g,l), (6) 

where = x ■ ■ • x '^t, where ^t is the family of functions from X* Z , 5f ^ = 
X ■ ■ ■ X ^ (T -times), where ^ is the family of functions from y x M. ^ X and ^'^^^ = 
=Sf X ■ ■ ■ X ^ ({T — l)-times), where =Sf is the family of functions from y x M. ^ M.. 

The problem belongs to the class of decentralized dynamic team problems with non- 
classical information structure. Such problems are difficult to solve as they are non- 
convex functional optimization problems. We can view the problem as a sequential 
stochastic optimization problem [HI E] by a fictitious partitioning of stage t into three 
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parts. The encoder transmits at , the decoder makes a decision at [t + i) and the 
memory is updated at (t + 1)~. Now we have a stochastic optimization problem with a 
horizon of 3T where the decision makers can be ordered in advance, thus the problem 
is sequential. Witsenhausen [8j presented a general framework to work with sequential 
stochastic optimization problems by converting them into standard form. The solution 
methodology presented therein is applicable only to finite horizon problems and can not 
be extended to infinite horizon problems. We exploit the structural results of |1| to 
obtain a solution methodology which can be extended to infinite horizon problems. For 
completeness of presentation we summarize the structural results of |4|| next. 

3 Structural Results 

Definition 1. Let be the encoder's belief about the memory contents of the receiver, 
i.e., 

Pm, (m) ^FT{Mt = m\X\Z\ c\ /*) . (7) 

For a particular realization x* and an arbitrary (but fixed) choice of c*, /*, the realization 
of PMt denoted by Pm„ is a PMF on Mj and belongs to , the space of PMFs owM. If 
X* is random vector and c*, /* are arbitrary (but fixed) functions, then Put is a random 
vector belonging to . 

Theorem 1 (Structure of Optimal Encoder). Consider the problem of minimizing 
the expected distortion given by ^ for any arbitrary (but fixed) decoder g and memory 
update I. Then, without loss in optimality, one can restrict attention to encoding rules of 
the form 

Zt = Ct{Xt,PM,_J, t = 2,...,T. (8) 

Theorem 2 (Structure of Optimal Decoder). Consider the problem of minimizing 
the expected distortion given by ^ for any arbitrary (but fixed) encoder c and memory 
update rule I. Then, the design of an optimal decoder is a filtering problem and an optimal 
decoding rule g* is given by 

xt = gtiyt^m-i) = rt{^tiyt,m~i)), (9) 

where 

it{y, m){x) = Vi{Xt = x\Yt = y, Mt-i = m) , (10) 

and 

Tti^t) = argmin ^ pt{x, x)Ct{x). (11) 

See j3j for a proof of these theorems. 
3.1 Implication of Structural Results 

The structural results simplify the problem as follows: 

(i) Theorem H] implies that at each stage t, without loss in optimality, we can restrict 
attention to encoders belonging to the family of functions from X x to 
Z. Thus, at each stage, we can restrict to optimizing over a fixed (rather than 
time-varying) domain. 
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(ii) Theorem [2l implies that the structure of an optimal decoders is a deterministic 
function of pt, the distortion measure at time t and ^t, the conditional PMF at 
time t, which in turn depends only on the choice of decision rules c* and Thus, 
an optimal decoder at time t can be written as = g^{c^,l^~^), implying that an 
optimal decoder obtained by Theorem [2l can be expressed in terms of the encoder 
and memory update rule as g*{c, I). For any design define 

J(c,0=J(c,/(c, /),/), (12) 
and consider the following problem: 

Problem 2. Under the assumptions of Problem^ choose a design {c*,l*) that is 
optimal with respect to the performance criterion of (|12|) . i.e., 




where = x ■ ■ ■ x ^5 (T -times) 

Clearly, J* = J* i.e., the design {c*,g*,l*) obtained by an optimal solution (c*,/*) 
of Problem O along with an optimal decoder g*{c*,l*) obtained by Theorem [2l is 
an optimal solution for Problem ^ 

In the next section we provide a sequential decomposition for Problem El 
4 Joint Optimization 

The critical step in obtaining an optimization methodology based on sequential decom- 
position is identifying an information state sufficient for performance evaluation of the 
system. In this section, we give expressions for an information state and explain how to 
obtain a sequential decomposition of the problem. The intuition behind our approach 
is as follows. The distortion at stage t depends on and Xt. We need to find a field 
basis and conditional basis for Xt (see ^) for each agent at each stage. However, just 
finding a field and conditional bases is not sufficient. These combined bases must form a 
state (in the sense of [10]) for the purpose of performance evaluation. Suppose ttj and (ft 
are the information states of the encoder and memory update respectively. They need 
to satisfy the following properties: 

(i) TTt is a function only of the encoder's information and the past encoding and memory 
update rules. Any choice of the present encoding rule q together with ttj determine 
ipt, the information state for the memory update at the next step. 

(ii) (pt is a function only of the receiver's information and the past encoding and memory 
update rules. Any choice of the present memory update rule k together with ipt 
determine tt^+i, the information state for the encoder at the next step. 

(iii) At each stage both the encoder and the receiver can evaluate the expected cost to 
go from their respective information state and choice of present and future deci- 
sion rules. This expectation is conditionally independent of the past decision rules, 
conditioned on the current information state. 

The above properties can be written more formally as follows: 
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(Sla) TTt is a function of x*, c*~^ and 
(Sib) (ft is a function of yt, rrit-i, c* and 
(S2a) ipt can be determined from vr^ and q. 
(S2b) TTi+i can be determined from tpt and /j. 

(S3) For the purpose of performance evaluation, ttj absorbs the effect of c*~^,/*"^ and 
V^i absorbs the effect of c*, on expected future distortion, i.e. 



s=t 



c,gj 



or alerntively 
(S3^) e\ pt{Xt,Xt 



[ s=t 
t s=t 



ft, k 



ft, Cf+i, 



= E^Pt{Xt,Xt 
e\ pt{Xt,Xt) 



Properties (S1),(S2),(S3) are equivalent to properties (S1),(S2),(S3*). (SI) and (S2) 
imply that TTf and ft are states and (S3) ensures that vr^ and ft absorb the effect of past 
decision rules on expected future distortion. Thus, they are sufficient for the purpose of 
performance evaluation. In this section we find information states vrt and ft that satisfy 
(S1)-(S3). We define the following: 

Definition 2. Let PvtMt-i be the encoder's belief about the channel output and memory 
contents of the receiver, i.e., 

PyuNU.Av. m) ^ Pr (F* = y, Mt.^ = m|X*, Z*, c\ l'-') . (14) 

For a particular realization and a particular choice c*, the realization of PYt,Mt-i, 
denoted by PYt,Mt-i, is a PMF on (Ft,Mt_i) and belongs to "P^x-^, the space of PMFs 
on 3^ X A^. If X* is a random vector and c*, are arbitrary (but fixed) functions, then 
PYt,Mt-i is a random vector belonging to 'py^-'^. 

Lemma 1. At each stage t, 

(i) there is a deterministic function z/((-) such that PYt,Mt-i = ^tiPhU-i, Zt), 

(a) there is a deterministic function ipt{-) such that Put = i^tiPvtMt-i^^t)- 

Proof See p]. □ 

Definition 3. Let H (resp. ^) be the space of probability measures on X x (resp. 
X X py^^j. Define vr^ and ft as follows: 



nt = Pi {Xt,PM,.,) , 

ft = Pr (Xt,Py,,Af,_i) , 



(15) 
(16) 



where Tit (resp. ft) belongs to 11 (resp. $j. 
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Theorem 3. tt^ and (pt are the information states for the encoder and memory update 
respectively, i.e., 

(i) there is a linear transformation Qt{ct) such that 

= Qtict)nt, (17) 

(ii) there is a linear transformation Qt{k) such that 

TTt+l = Qtik)Vt, (18) 

(Hi) for any choice of c and I, the expected conditional instantaneous cost can be ex- 
pressed as 

E{ptiXt,Xt)\c\g;ic\r')j'-' }=ptiipt). (19) 

where g*{c^J^~^) is an optimal decoding rule corresponding to c*, Z*^^ and pt{-) is a 
deterministic function. 

Proof. This follows from Lemma [H See P] for detailed proof. □ 

The choice of functions c*, Z*^\ g^{c^,l^^^) make the variable Xt a random variable 
with well defined distribution. Thus, the performance criterion of (fT2|l can be rewritten 

as 



t=i 



c, g*{c, 0, / [ = 5] E { ptiX,, X,) I c*, g;{c\ I'-'), I'-' } 
J t=i 

T 

= T.Pti^t). (20) 



t=l 

Notice that (fT7|l and (|T8|l imply that tt^ and tpt are states, i.e. they satisfy (SI) and (S2). 
Moreover, (fT!Hl and (f^ imply that ttj and ipt are sufficient for performance evaluation, 
i.e., satisfy (S3). Hence Theorem El implies that Problem [21 is equivalent to the following 
deterministic optimization problem: 

Problem 3. Consider a deterministic system which evolves as follows: 

= Qt{Ct)TTt, t=l,...,T, (21) 

-n^t+i = Qt{k)^t, t = l,...,T-l, (22) 

where Ct and It are functions belonging to and ^ respectively and Qti') and Qti,-) are 
deterministic transforms depending on Ct and It respectively. The initial state tti of the 
system is known. If the system is in state ip at stage t, it incurs a cost pt{^)- 

The optimization problem is to obtain decision rules c = (ci, . . . , ct), I = (h, ■ ■ ■ , h-i) 
to minimize the total cost over horizon T, i.e., find optimal design (c*,Z*) such that 

J{c*,n = J*= mf J: pticpt). (23) 

t=i 
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This is a classical deterministic control problem; optimal functions (c*,/*) are deter- 
mined by the nested optimality equations given below. 

Theorem 4. An optimal design (c*,/*) for Prohlem\^ (and consequently for Problem\B 
and thereby for Problem QJj can be determined by the solution of the following nested 
optimality equations: 



Vt{nt) 



0, 
inf 



mm 



Pt{Qtict)7r)+Vt{Qtict)7r) 



T 



Proof. This is a standard result, see [Sl Chapter 2] 



(24) 
(25) 

T-l. (26) 
□ 



5 Time Homogeneous System — Finite Horizon Case 

For many applications the system is time-homogeneous, that is, the source is time- 
invariant Markov process {Pxt+i\Xt does not depend on t), the channel is time-invariant 
(transition matrix Pt does not depend on t) and the distortion metric pt{-) is time invari- 
ant. For such a system, the functions i^t{-), i'ti'), the linear transforms Qt(-), Qt{-) and 
the distortion pt{-) defined in Theorem [HI are time-invariant, so we can drop the subscripts 
t and simply refer them as i/(-), Q{-), Q{-) and p(-) respectively. So, we obtain an 
equivalent of Theorem El making the corresponding changes. Thus, Problem El reduces to 
a time-homogeneous problem — one in which state space, action space, system update 
equation and instantaneous cost do not depend on time. Hence the optimality equations 
of Theorem (HI can be written in a more compact manner We can define the following: 

Definition 4. LetV (resp. V) be the family of functions from H (resp. ^) to E"^. Define 
operators W{c) (resp. W{1) ) from VtoV (resp. VtoV)as follows: 

{W{c)V){'k)=p{Q{c)t,)+V{Q{c)tx), (27) 
{W{l)V){^) = V{Q{l)v). (28) 

Further define transformations W (resp. W) from V to V (resp. VtoV) as follows: 

(W)(vr)= inf {Wic)V)in), (29) 

{WV){^) = M{W{l)V){^). (30) 



Theorem 5. For the time-homogeneous case, the value functions Vt andVt of Theorem^ 
evolve in a time-homogeneous manner as follows: 



Vt = WVt+i, 
Vt = Wt, 
with the terminal condition given by 



t = 
t = 

Vt = 0. 



T-l 
T 



(31) 
(32) 

(33) 



The arguments minimizing Vt and Vt at each stage determine the decision rules Ct and k. 
Proof. This follows immediately from Theorem HI □ 
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6 Time Homogeneous System — Infinite Horizon Case 

We consider a time-homogeneous system as in Section [HI However, instead of a finite 
horizon T, we consider the infinite horizon case with performance of a design determined 
by 



J{c,g,l) = E\ Y,P'''piXt,Xt) 
t=i 



c,gj L (34) 



where /3 G (0, 1) is called the discount factor. With a slight modification of the proof 
of lU Section 2.4] one can show that the structural results of Section El are also valid in 
this case. Further, Theorem El (with the changes mentioned in previous section) holds 
for the infinite-horizon case also. 

Definition 5. Define operators W , W and transforms W, W as in Definition^ with 
one change — modify the definition of W to take the discounting into account as follows: 

(l^(c)y)(7r) = ~p{Q{c)ti)+(3V{Q{c)'k). (35) 

Theorem 6. For the infinite horizon time-homogeneous system with the performance 
criterion of JSH); the evolution of value function is governed by the following set of 
equations 

Vt = W% (36) 
Vt = WVt+i. (37) 

The arguments that minimize Vt and Vt+i at each stage determine the decision rules Ct 
and It. 

Proof. This is the solution of the time-homogeneous problem formulated by considering a 
time-homogeneous version of Problem (HI with the optimization criteria being minimizing 
^{ET=if^'-'piv't)\c,g*ic,l)J}. □ 

Definition 6. A design (c, /), c = (ci,C2,...), / = (/i,/2i---) is called stationary (or 
time-invariant) if Ci = C2 = ■ ■ ■ = c, li = I2 = ■ ■ ■ = I . 

Theorem 7. For the time homogeneous case with the performance measure of pijl . if 
the distortion measure p(-) is bounded and discount factor P < I, then stationary designs 
are e-optimal, that is, for any design (c', /') and any e > 0, there exists a stationary 
design (c°°,/°°) such that 

J{c'^,r) = V{n,)<J{c',l')+e, (38) 

where V is the unique fixed point of 

V={WoW)V, (39) 

and c and I are the corresponding argmin and c°° = (c, c, . . . ), l°° = (/,/,...). 

Proof See 0. □ 

We have shown that a unique stationary e-optimal policy exists. Thus, for the infi- 
nite horizon problem, without loss of optimality, we can restrict attention to stationary 
policies. This simplifies the implementation of an optimal policy. 
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7 Discussion 

It was shown in ^ that all sequential problems can be transformed to a standard form by 
moving all the uncertainty to the first stage and at each stage augmenting the state vari- 
able to carry all the information needed to determine the cost. Further an optimal policy 
for a problem in standard form can be obtained by solving a deterministic optimiza- 
tion problem. We believe that our methodology has a similar spirit as Witsenhausen's 
standard form. We have a decentralized optimization problem that is sequential and an 
optimal design is obtained by the solution of a deterministic optimization problem. In 
our solution the state space is not increasing with time and allows us to use our approach 
to infinite horizon problems while the standard form is applicable only to finite horizon 
problems. The structural results of [IJ are critical to our approach as they allow us to 
obtain an information state whose dimensionality does not change with time. 

8 Conclusion 

We have developed a methodology for the determination of jointly optimal real-time 
encoding, decoding and memory update strategies for point-to-point communication sys- 
tem. This methodology has been extended to k-th oder Markov sources, distortion metric 
accepting a finite delay of S units and channels with memory (see jS| for details). We 
believe that the same methodology can be used for the determination of jointly optimal 
real-time encoding, decoding and memory update strategies for more complex communi- 
cation systems. 
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